CTT and IRT 1 Test Construction using CTT and IRT with Unrepresentative Samples

We compare test construction using CTT and IRT in several sample sizes (from N=20 to N=5000) and degrees of representativeness (represented by selecting the top 20%, 40%, 60%, 80% or 100% of a population) using a Monte-Carlo simulation design. Little support was found for our hypothesis that IRT would outperform CTT in building informative tests, especially in large or unrepresentative samples. The test construction algorithm was more influential than sample size or representativeness; picking the most discriminating items from across the range of difficulty slightly favored CTT, while IRT was slightly favored when the IRT test construction algorithm directly maximized the evaluation criteria. We conclude that test construction using either CTT or IRT produces empirically similar exams and IRT is only preferred when there is a target test information function.